Skip to content

feat: make reboot — diagnose + auto-heal the lab after a host reboot - #17

Merged
leeclay95 merged 1 commit into
mainfrom
feat/make-reboot-self-heal
Jul 11, 2026
Merged

leeclay95 merged 1 commit into
mainfrom
feat/make-reboot-self-heal

Conversation

@leeclay95

Copy link
Copy Markdown
Owner

Why

A fresh Ubuntu reproducibility test surfaced that a host reboot leaves the lab broken in ways kubectl get pods doesn't explain, and the failures compound:

Symptom Cause
failed to setup network ... /run/flannel/subnet.env flannel subnet file on tmpfs, wiped on node restart
ESO SecretSyncedError, lookup host.k3d.internal ... i/o timeout k3s strips host.k3d.internal from CoreDNS on restart
webapp ImagePullBackOff, dial tcp 127.0.0.1:5100: connection refused image dropped from node containerd; node can't pull the floci ECR ref

Plus a distinct root cause for the image never landing on modern Docker: the containerd-snapshotter image store (default on recent Docker/Ubuntu) makes k3d image import silently no-op — docker save emits an OCI layout the importer rejects (content digest not found) while reporting success.

What

  • make reboot (scripts/recover.sh) — diagnoses and auto-heals, in order: floci up → cluster started → flannel (bounces node if subnet.env missing) → host.k3d.internal → webapp image → ESO re-sync → log shipper → webapp rollout → status + app HTTP code. Idempotent; never touches Terraform state, floci secrets, or infra/secrets roots.
  • scripts/ensure-webapp-image.sh — snapshotter-safe image loader: verifies the image actually landed on the node and falls back to pulling the identical upstream image into the node's containerd + retagging. Used by both make reboot and the deploy.
  • terraform/cluster/webapp.tf — image import now calls the loader, so a fresh make deploy works regardless of Docker's storage driver.
  • README — "After a reboot" section + image-loading note.

Verified

Ran make reboot on a live cluster: healthy checks pass, host-fix reapplied, ESO SecretSynced, webapp 1/1 Running, app HTTP 200. Security gate (kubesec/tfsec/conftest) PASS.

🤖 Generated with Claude Code

A host reboot restarts the k3d node container and breaks several things that a
plain `kubectl get pods` won't explain, and they compound:
- flannel /run/flannel/subnet.env wiped (tmpfs) -> pod sandbox failures
- host.k3d.internal stripped from CoreDNS -> ESO can't reach floci
- webapp image drops out of node containerd -> ImagePullBackOff (node can't pull
  the floci ECR ref: inside the node it resolves to the node's own loopback)
- floci down / lost un-flushed data

New:
- scripts/recover.sh (make reboot): checks and fixes each of the above in order,
  idempotent, never touches Terraform state / floci secrets / infra roots.
- scripts/ensure-webapp-image.sh: snapshotter-safe image loader. `k3d image
  import` silently no-ops on Docker's containerd-snapshotter store (default on
  recent Docker/Ubuntu) — docker save emits an OCI layout the importer rejects
  ("content digest not found") but reports success. This verifies the image
  landed and falls back to pulling the identical upstream image into the node's
  containerd + retagging. Used by both `make reboot` and the deploy.
- terraform/cluster/webapp.tf: image import now calls ensure-webapp-image.sh, so
  a fresh `make deploy` works regardless of Docker's storage driver.
- README: "After a reboot" section + the image-loading note.

Validated end-to-end on a live cluster: healthy checks pass, host-fix reapplied,
ESO re-synced (SecretSynced), webapp 1/1, app HTTP 200. Security gate PASS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@leeclay95
leeclay95 merged commit de304f7 into main Jul 11, 2026
1 check passed
@leeclay95
leeclay95 deleted the feat/make-reboot-self-heal branch July 11, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant